Testing alarm-based earthquake predictions

S U M M A R Y Motivated by a recent resurgence in earthquake predictability research, we present a method for testing alarm-based earthquake predictions. The testing method is based on the Molchan diagram—a plot of miss rate and fraction of space–time occupied by alarm—and is applicable to a wide class of predictions, including probabilistic earthquake forecasts varying in space, time, and magnitude. A single alarm can be simply tested using the cumulative binomial distribution. Here we consider the more interesting case of a function from which a continuum of well-ordered alarms can be derived. For such an ‘alarm function’ we construct a cumulative performance measure, the area skill score, based on the normalized area above a trajectory on the Molchan diagram. A score of unity indicates perfect skill, a score of zero indicates perfect non-skill, and the expected score for a random alarm function is /2. The area skill score quantifies the performance of an arbitrary alarm function relative to a reference model. To illustrate the testing method, we consider the 10-yr experiment by J. Rundle and others to predict M ≥ 5 earthquakes in California. We test forecasts from three models: relative intensity (RI), a simple spatial clustering model constructed using only smoothed historical seismicity; pattern informatics (PI), a model that aims to capture seismicity dynamics by pattern recognition; and the U. S. Geological Survey National Seismic Hazard Map (NSHM), a model that comprises smoothed historical seismicity, zones of ‘background’ seismicity, and explicit fault information. Results show that neither PI nor NSHM provide significant performance gain relative to the RI reference model. We suggest that our testing method can be used to evaluate future experiments in the Collaboratory for the Study of Earthquake Predictability and to iteratively improve reference models for earthquake prediction hypothesis testing.